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AVERAGING  METHOD  FOR  STOCHASTIC  APPROXIMATIONS 
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Abstract 

Stochastic  approximations  with  constant  gain  coefficients  and  dependent  noise 
and  nonlinear  or  even  ^discontinuous  dynamics'^  have  many  applications  in  control, 
automata  and  communication  theory.  When  the  gain  coefficient  is  small,  an  asymp¬ 
totic  theory  is  developed  which  gives  much  information  on  the  character  of  the 
paths  and  errors.  The  method  involves  both  Vaveraging^  and  '’stability^  ideas. 

The  ideas  are  outlined.  An  example  which  illustrates  the  basic  ideas  and  tech¬ 
niques  is  given. 
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I.  Introduction 

Several  quite  powerful  methods  are  available  for  analyzing  asymptotic  prop¬ 
erties  of  many  kinds  of  stochastic  approximations  with  gain  sequences  which  are 
either  constants,  or  tend  to  zero.  Methods  for  the  first  type  of  gain  sequence  - 
examples  of  which  abound  -  are  much  less  well  known.  In  this  paper,  we  develop 
a  method  for  such  a  problem,  based  on  "stability",  "averaging"  and  "diffusion 
approximations  via  weak  convergence  theory"  -  ideas  which  have  served  very  well 
in  many  other  areas.  The  techniques  have  great  power  and  applicability.  The 
general  idea  will  be  outlined  and  a  simple  example  dealt  with  in  detail.  Despite 
the  simplicity  of  the  example,  it  well  illustrates  the  general  approach,  the  kind 
of  calculations  which  need  to  be  done  and  the  type  of  results  which  can  be 
expected.  The  example  is  used  simply  as  a  vehicle  for  explaining  the  main  ideas. 
Nevertheless,  only  the  surface  of  a  large  subject  will  be  touched.  There  is 
much  work  on  the  problem  (when  the  gain  sequence  tends  to  zero)  of  the  asymptotic 
properties  of  stochastic  approximation  via  the  study  of  the  stability  of  an  assoc 
iated  ordinary  differential  equation  [1},  121.  The  local  asymptotic  behavior 
(near  the  limit  points) ,  on  the  other  hand,  is  obtained  via  the  study  of  an  assoc 
iated  stochastic  differential  equation  (as  in  rate  of  convergence  studies  ID, 

[3],  [4]).  Here  similar  intentions  are  pursued  for  process  with  constant  (but 
small)  gains,  state-dependent  noise,  and  perhaps  discontinuous  forcing  functions. 

We  are  concerned  with  asymptotic  properties  (n  -*  «,  then  e  -►  0)  of  a  sub¬ 
class  of  vector  stochastic  difference  equations  of  the  form  i  -j — - 

J  Accession  Ter 


(1.1)  y‘  =  Y*  +  eh  (vSd)  +  St  g.(Y^,^)  +  o(e)  , 
n+i  n  c  n  n  e  n  n 


k 


Euclidean  r-space, 


2. 


where  hE  and  g^  are  not  necessarily  continuous  everywhere  and  is  a  sequence 

of  correlated  (and  perhaps  {YE} -dependent)  random  variables,  for  each  e  >  0. 

n 

Equations  such  as  (1.1)  occur  very  frequently  in  applications  in  stochastic  con¬ 
trol,  communication  and  automata  theory.  Often  the  g^  or  h£  are  indicator 

functions;  i.e.,  the  iterate  YE  moves  "left"  or  "right"  by  e,  depending  only  on 

n 

whether  some  event  or  other  occurred. 

There  is  a  vast  (stochastic  approximation)  literature  on  (1.1)  when  e  is 

replaced  by  a  sequence  e  -*-0.  But,  very  frequently  in  implementations  of 
'  n 

stochastic  approximation,  is  either  held  constant  or  else  H  >  0  as  n  +  ®, 
owing  to  the  desire  to  track  changes  or  for  the  purpose  of  improving  robustness. 
Consider  a  particular  case  of  (1.1),  a  scalar  Robbins-Monro  procedure  of  the  form 

(1.2)  X*  -  Xe  -  e  sign[k(XE)+d), 
n  +i  n  n  n 

E  E 

where  £  has  a  symmetric  distribution,  {£  }  are  not  necessarily  independent  (and 
n  n 

might  even  depend  on  the  iterates),  k(*)  is  continuous,  and  there  is  a  0  such 

that  k(x)  >  0  for  x  >  0,  and  k(x)  <  0  for  x  <  0.  In  particular,  suppose  that 

for  x  >  0  (resp.,  x  _<  0) ,  k(x)  is  bounded  below  (above,  resp.)  by  an  increasing 

function  which  has  a  non-zero  slope  near  the  origin.  We  might  want  to  prove  that 
P 

XE  •*  8  as  n  -*■  «*  and  e  -*•  0,  or  even  to  get  a  good  idea  of  the  statistical  struc¬ 
ture  of  the  tail  of  the  sequence  {XE-0}.  Define  UE  ■  (XE-0)//e.  Something  akin 

n  n  n 

to  a  rate  of  convergence  can  be  obtained  by  studying  the  asymptotic  properties 

of  {UE}  -  which,  as  we'll  see,  brings  us  back  to  a  process  of  the  form  (1.1)  with 
n 

a  /e  term  included. 

There  are  numerous  other  applications  of  (1.1)  -  particularly  to  systems 
whose  dynamics  are  determined  by  "logical"  criteria  -  where  the  movement  is 


3. 


determined  partly  by  the  satisfaction  of  a  "logical"  criterion.  Such  systems 
have  not  received  much  attention,  important  though  they  are.  One  very  clever 
approach  and  some  nice  applications  appear  in  [5],  [6].  Although  that  method  and 
ours  both  exploit  "averaging"  phenomena,  our  method  is  different.  We  are  not 
confined  to  finite  intervals  [n:  en<T]  for  some  arbitrary  T  (indeed,  it  is  the 
"tail"  which  is  of  most  interest  here) ,  and  [5]  and  [6]  did  not  explicitly  deal 

with  a  /e  factor  in  (1.1).  The  /e  factor  always  occurs  when  a  local  analysis 

£ 

(involving  U  )  is  done.  When  this  factor  is  present  the  limiting  determining 
n 

equation  is  a  stochastic  rather  than  ordinary  differential  equation.  We  concen¬ 
trate  on  the  asymptotic  part  of  (X^}  or  (u^},  for  sma^^  E'  often  the  part  of 
greatest  interest.  One  rather  direct  but  useful  method  for  the  asymptotic  problem 
in  in  [4] . 

In  practice  e  is  often  small.  But,  generally,  it  is  very  hard  to  get  infor¬ 
mation  on  what  happens  when  e  is  not  small,  just  as  in  stochastic  approximation 
it  is  hard  to  get  information  on  the  theoretical  behavior  when  n  is  not  large. 

A  main  question  is  how  well  the  theory  for  small  e  predicts  what  happens  in  other 
cases.  Simulations  on  continuous  parameter  problems  which  resemble  these  in 
certain  respects  indicate  that  the  prediction  is  good  for  many  cases  when  the 
parameter  e  is  in  a  "normal"  range.  In  some  cases,  the  asymptotic  behavior  is 
better  than  predicted.  Generally,  the  closeness  of  the  prediction  to  the  behavior 
actually  observed  seems  to  depend  heavily  on  the  form  of  the  nonlinearities,  and 
on  the  correlation  structure  of  the  noise,  and  generalizations  are  hard  to  make. 

By  concentrating  on  large  n,  we  are  effectively  concentrating  on  what  happens  after 
the  "transient"  period  is  over. 

The  techniques  used  here  come  from  references  [7],  [8],  [11],  which  concern  the 
general  problem  (1.1)  (or  continuous  time  analogues)  under  various  sets  of 
assumptions.  The  emphasis  in  [7]  is  on  the  case  where  g^,  h^  are  smooth,  but 
an  outline  is  given  of  the  method  for  the  more  general  case.  Reference  [8] 

_  il 
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treats  two  problems  in  great  detail,  one  arising  in  an  automata  approach  to  tele¬ 
phone  traffic  routing,  and  the  other  a  recursive  quantizer  in  communication 
theory.  Our  purpose  here  is  to  describe  the  general  method,  citing  results  from 
17],  [8]  and  avoiding  duplication  of  proofs  wherever  convenient.  As  an  illustra¬ 
tion  of  the  power  and  usefulness  of  the  method,  a  detailed  analysis  of  (1.2) 

(for  both  the  tail  of  {XE}  and  {UE})  will  be  given.  The  development  will  illustrate 

n  n 

the  usefulness  of  the  approach  for  other  problems.  The  case  (1.2)  is  simple  (but 
not  trivial),  but  a  very  similar  method  of  analysis  is  used  in  more  general  cases. 


Outline  of  the  paper.  The  first  important  result  (Theorem  2)  concerns  the 

tightness  of  {Ufi,  large  n,  small  e}.  By  this  we  mean  that  there  is  an  eQ  >  0, 

whose  value  is  not  important,  such  that  for  each  e  £  eQ,  there  is  an  integer 

N  <«  such  that 
e 

(1.3)  lim  sup  P{|ue|  >  K}  =  0. 

n  — 

K-*»  e<eQ 
n>Ne 

I.e.,  {UE,  large  n,  small  e}  is  bounded  in  probability.  Such  a  result  makes  pos¬ 
sible  a  detailed  asymptotic  analysis  of  (UE),  since  it  implies  that  the  "tails" 
of  {X^}  ate  uniformly  close  to  9  in  a  specific  statistical  sense.  To  simplify 
the  development  most  details  are  for  the  special  case  (1.2).  More  general  cases 
can  readily  be  dealt  with  in  a  very  similar  way,  at  the  expense  of  a  somewhat 
heavier  notation,  but  the  simple  case  illustrates  the  main  ideas  and  methods. 

Next,  we  define  a  continuous  parameter  process  Ue(-).  The  {n ^ }  will  always 
satisfy  (1.3).  Let  0^}  denote  any  sequence  of  integers  satisfying  n^  NE  a*1*3 
n£  "  as  c  -*■  0.  Their  values  will  either  be  stated  when  needed  or  will  be 


5. 

C  G  C 

unimportant.  Define  the  process  U  (t)  =  U  ..  on  [ie,ie+e) .  Define  Y  (•)  by 

ne+i 

Ye(0)  *  YE  and  YE(t)  =  YE  on  [en,en+e)  (refer  to  (1.1)).  The  general  result  is 
u  n 

G  IT 

that  U  (•)  converges  (under  suitable  conditions)  weakly  in  D  [0,°»)  to  a  Gauss- 
Markov  diffusion  U(*)  which  satisfies  an  equation  of  the  form 

(1.4)  dU  =  -GUdt  +  cdB, 

where  B ( - )  is  a  standard  Browninan  motion  and  G  >  0.  If  ®  fast  enough  as 

e  -*■  0,  then  the  limit  U(-)  is  stationary.  All  terms,  conditions  and  parameters 

will  be  defined  below,  but  first,  let  us  examine  the  limit  (1.4). 

G  G  G 

For  small  e,  the  processes  {X  ,U  }  (or  {Y  },  for  the  case  (1.1))  have  a 
r  n  n  n 

(perhaps  long)  transient  period,  before  their  distributions  "settle  down"  - 
£ 

especially  if  (XQ-0)  is  large  compared  to  e.  We  are  concerned  only  with  the 
asymptotic  part  -  after  the  so-called  transient  period  is  over;  i.e.  when  UE 
0(1).  We  can  get  the  variance  of  the  asymptotic  part  from  (1.4),  since  XE  'v, 
ft  UE+0.  Also,  (1.4)  gives  the  local  correlation  structure  of  the  asymptotic  part 
of  the  process.  Weak  convergence  methods  are  a  very  natural  and  convenient  tool 
for  getting  our  results.  The  details  of  the  derivation  of  (1.4)  will  be  given 
for  the  model  (1.2).  Very  similar  methods  can  be  used  for  more  complex  problems 
17],  18].  Next,  some  comments  and  definitions  concerned  with  weak  convergence 
theory  will  be  given.  Then  we  comment  on  the  so-called  martingale  problem  of 
Strook  and  Varadhan  (9],  which  provides  a  characterization  of  the  desired  limit 
process  which  is  convenient  from  the  point  of  view  of  simplifying  the  proof  of 
showing  that  it  actually  is  the  limit.  Section  II  contains  the  main  background 
theorems.  For  (1.4),  tightness  of  {UE,  large  n,  small  e)  is  proved  in  Section 
III,  and  in  Section  IV  we  show  how  to  get  (1.4)  for  the  case  of  model  (1.2) . 
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A  note  on  weak  convergence  theory.  Only  a  few  comments  will  be  made.  For 
full  details  see  [10],  or  [1],  Chapter  3,  for  a  brief  summary.  D  [0,®)  denotes 
the  space  of  Revalued  functions  on  [0,“)  which  are  right  continuous  and  have 
left-hand  limits.  The  "process"  UE(*)  is  treated  as  a  random  variable  defined 
on  the  sample  space  D  [0,“)  and  induces  a  measure  on  it  which  we  denote  by  P£ 

(a  useful  topology,  called  the  "Skorokhod"  topology,  is  used  on  D  ;  see  [10]). 

{u  (•)}  is  said  to  be  tight  iff  for  each  6  >  0  there  is  a  compact  set  D  [0,®) 

such  that  P  {K^}  1-6,  all  e.  UE(0  is  said  to  converge  weakly  to  U(*)  if  U(-) 

has  paths  in  Dr[0,°°),  and  induces  a  measure  P  on  it,  and  for  each  real-valued 
continuous  function  F(-)  on  Dr[0,®),  / F(v)dP£(v)  / F(v)dP(v)  as  e  +  0.  The 

basic  result  is:  I£  (uE(-))  is  tight  on  Dr[0,®),  then  each  subsequence  contains 
a  further  subsequence  which  converges  weakly  to  some  process  with  paths  in 
Dr[0,®) .  Our  job  here  is  to  characterize  the  limit  process  and  to  show  that  it 
does  not  depend  on  the  subsequence.  Thus  weak  convergence  is  a  substantial 
extension  of  convergence  in  distribution.  It  is  a  tool  that  is  extremely  useful 
in  many  areas  of  applied  probability  where  limit  or  approximation  problems  are 
of  concern.  Criteria  for  tightness  and  weak  convergence  are  often  given  in 
terms  of  the  multivariate  distributions  of  the  processes  {Ue(*)}.  See  Section  II. 

A  note  on  the  martingale  problem.  This  problem  arises  because  in  the  weak 
convergence  analysis  it  is  convenient  to  characterize  the  limit  U('),  whatever 
it  may  be,  by  showing  that  it  solves  a  certain  set  of  equations  which  are  known 
as  the  martingale  problem.  Then  we  show  that  the  U(*)  of  (1.4)  is  the  only 
solution  to  that  martingale  problem.  Let  x(0  be  the  solution  to  the  stochastic 
differential  equation  (SDE) 

fit  j. 

(1.5)  dx  ■  b(x,t)v+  o(x,t)dB,  B (•  J  *  standard  Brownian  motion  in  R  , 


and  set  o(x)a'(x)/ 2  =  a(x) .  Suppose  that  b(*,*)  and  a ( * , - )  are  continuous  in 

(x,t)  and  satisfy  a  uniform  Lipschitz  and  linear  growth  condition  (in  x) .  Then 

it  is  well  known  that  the  SDE  (1.5)  has  a  unique  solution  in  the  It6  sense, 
t 

Since  /o(x  )dB  is  a  martingale,  for  any  smooth  bounded  function  f(*),  lt6's 
0  s  s 

lemma  implies  that 

t 

(1.6)  f(x(t),t)  -  f  (x(0)  ,0)  -  j  (A  +  1^-)  f  (x(s)  ,s)  ds  =  Mf(t) 

0 

is  a  martingale  for  each  initial  condition  x(0)  =  x,  where 


(1.7)  A  =  [  b.(x,t)  +  l  a  (x,t) 

l  l  1,3  J  ID 

=  differential  generator  of  (1.5). 


3T 

Let  y(-)  denote  the  generic  element  of  D  ,  let  denote  the  contin- 

uous  functions  on  Rr  x  [0 ,«•)  with  compact  support  and  letif^'^  denote  the  subclass 

^1  2 

whose  mixed  a  t-derivatives  and  6  x-derivatives  are  continuous.  For  f £  ,  define 

t 

(1.8)  T^Ct)  -  f(y(t),t)  -  f  (y  (0)  ,0)  -  j  (A  +  |h  f  (y(s)  ,s)ds, 

0 

where  A  is  an  operator  of  the  form  (1.7)  -  but  not  necessarily  satisfying  the 
Lipschitz  and  growth  conditions.  If  for  each  x,  there  is  a  measure  on  Dr[0,°>>) 
such  that  Px.{y(0>*x>  »  1  and  Mf<*)  is  a  P ^-martingale  for  each  f£^'2,  then  {P^}  is 
said  to  solve  the  martingale  problem  (of  Strook  and  Varadhan  (9)).  If  P^  is 
unique  for  each  x  (as  it  is  under  the  Lipschitz  and  growth  condition  cited  above) , 
then  the  {Px>  induce  a  Markov  process  on  the  sample  space  Dr[0,<»).  Also,  for 


each  x,  there  is  a  B(0  such  that  under  p^,  the  process  satisfies  (1.5)  with 
initial  condition  x. 

The  martingale  formulation  of  SDEs  is  very  useful  in  convergence  studies. 

The  basic  background  theorems  for  our  case  [7],  [11]  are  proved  by  first  showing  tight- 

ness  of  (U  (•)}  and  then  showing  that  M^(-)  is  a  martingale  for  each  f  £^0' 

when  any  weak  limit  U(*)  is  substituted  for  y(*)  and  P^  is  replaced  by  the  measure 
% 

of  U(*)  and  A  is  the  operator  (1.7)  (in  our  case  this  specializes  to  the 
operator  of  (1.4)).  Below,  an  assumption  concerning  uniqueness  of  the  solution 
of  the  martingale  problem  will  be  made.  This  is  needed,  for  otherwise  the  weak 
limits  of  (Ue(.))  might  not  be  unique. 

Truncation.  For  mathematical  as  well  as  practical  reasons,  it  is  helpful 
to  work  with  a  truncated  {x^}.  The  particular  technical  difficulty  introduced 
in  the  untruncated  case  will  be  discussed  after  the  proof  of  Theorem  2 .  The 
truncation  introduced  now  alters  the  basic  problem  (1.2)  and  has  nothing  to  do 
with  the  "technical"  truncation  introduced  in  the  first  paragraph  of  Sec¬ 

tion  II.  Suppose  that  we  know  real  numbers  x.,x  such  that  0  £  (x. ,x  ) .  Let 
the  bar  |  denote  truncation.  Then  we  replace  (1.2)  by 

Ix 

U. 

X£ 

liquation  (1.9)  is  the  actual  algorithm  whose  asymptotic  properties  are  to  be 

studied.  There  are  continuous  functions  a  (•),  b  (•)  such  that  a  (x)  =  b  (x)  = 

'  G  C  G  G 

e  in  [x^+e,xu~e]  and  both  functions  have  values  in  [0,e]  and  are  infinitely 
differentiable  functions  of  x  except  possibly  at  the  points  x^+e,  xu-e'  and 
they  are  such  that 


9. 


(1.10)  X*  =  XG  -  a  (Xe)  l{k(XG)+£G>0)  +  b  (XG)  l(k(XG)+£G<0} , 
n+1  n  e  n  n  n—  e  n  n  n 


where  I { * }  is  the  indicator  function. 


II.  Weak  Convergence  Theorems 

Notation  and  a  comment  on  tightness .  The  theorem  below,  taken  from  [7], 
assumes  tightness  of  {Y  (•)}  (or  (U  (■)}).  Such  tightness  is  not  hard  to  get 
under  reasonable  conditions  by  Theorem  2  of  [7]  if  { YG ( • ) >  (or  { ue ( - ) } )  is 
bounded.  This  will  be  further  commented  on  below.  We  can  bound  the  processes 
by  a  truncation  device  which,  since  it  is  used  only  as  a  technical  tool,  does 
not  lose  us  any  generality,  and  it  will  now  be  explained.  Loosely  speaking, 
if  the  truncated  processes  defined  below  exhibit  a  suitable  weak  convergence, 
then  so  do  the  (UG(-)}  and  { YG ( - ) }  as  originally  defined.  Define  SN  = 

{x-;  I x|  <N} ,  and  let  b  (x)  be  a  function  with  values  in  [0,1],  equal  to  unity  in 

ii —  N 

E  N 

S„,  equal  to  zero  out  of  S.T1,  and  infinitely  differentiable.  Define  Y  '  (and 
N  N+I  n 

for  (1.2),  UG'N)  by 


(2.1)  YG'"  =  YG,N  +  [eh  (YG'N,5G)  +  Ji  g  (YG'N,^)  +  o(e)  ]b  (YG'N)  , 
n+1  n  enn  Enn  Nn 


(2.2)  UG'”  =  UG,N  -  v'e  sign[k(XG'N)+eG]b  (UE,N), 
n+1  n  n  n  N  n 


xG,N  =  Jz  uG,N  +  e. 

n  n 


Let  YG'N(.)  and  UG,N(’)  denote  the  corresponding  interpolations  of  and 

{UG,N}.  We  use  UG,N(0)  =  UG  if  |uG  I  <  N,  and  set  it  equal  to  zero  otherwise, 
n  N  N 

N  e  e 

Similarly  for  Ye,N(0) .  Owing  to  the  tightness  of  this  causes  no  problem. 

We  will  see  below  that  (2.2)  is  consistent  with  (1.9)  -  (1.10)  in  that  the 
weak  limits  of  UG(*),  where  UG(*)  is  obtained  from  (1.9)  -  (1.10),  are  the  same 


10. 


as  those  which  Theorem  1  below  gives  us  for  (2.2)  ,  as  e  -*•  <*,  then'N 

Suppose  for  the  moment  that  for  each  N,  {yE,N(*)}  or  (UE'N(*)}  converges 

N  N 

weakly  to  U  (•),  a  diffusion  process  (with  differential  operator  denoted  by  A  ) 

whose  part  up  to  first  escape  from  equals  the  part  of  U(-)  up  to  first  escape 

from  SN>  Then  if  the  solution  to  the  martingale  problem  for  operator  A  (A  being  the 

differential  generator  of  U(*))  is  unique,  UE(0  •+  U(')  weakly.  Let  EE,N  denote 

expectation  conditioned  on  (ye,N,  i<n,  i<n)  or  on  {xE,N,  i£n,  £?,i<n}, 

£ 

depending  on  the  case  (2.1)  or  (2.2).  E^  is  similarly  defined  when  the  superscript 
N  is  absent. 

The  following  is  an  adaptation  of  Theorem  1  of  (7]  to  our  case.  It  is 
stated  for  the  general  case  (1.1),  (2.1). 

Theorem  1.  Let  the  differential  generators  A  and  AN  have  continuous  coeffic¬ 
ients,  which  are  equal  in  SN  for  each  N.  Let  the  solution  to  the  martingale 
problem  corresponding  to  operator  A  be  unique  on  Dr[0,“>) ,  for  each  x  E  Rr.  For 
each  N  and  f  ( * ,  • )  E  a  dense  set  (sup  norm)  in  ,  let  there  be  a  sequence 
{f  '  (’)}  of  random  functions  satisfying  the  following  conditions.  Each  is 
constant  on  each  (ne,ne+e)  interval,  at  ne  it  is  measurable  with  respect  to  the 
o-algebra  induced  by  (ye,N,  j<n,  ^ ,  j<n}  and  for  each  N  oyd  V 

(2.3)  sup  E|fe,N(en)  |  +  sup  i  e| E£ ,Nf E ,N (sn+e) -f E ,N (en)  |  <  =°, 

n,c  n,e  ‘ 

(2.4)  E  j  f E ,N(en) -f  (YE  ,N, en)  I  -*■  0  as  e  -*  0  and  en  =  t, 

i  n  1 

„e,N  e,N,  ,  ,e,N,  . 

E  f  ( En+e) -f  (£n) 

(2.5)  E| - - - (—  +  A  )f  (Yn'  ,en)  |  -*■  0  as  e  +  0  and  en  =  t. 

Then  if  (ye,N(-)  ,  0<e_<e0}  is  tight  on  Dr  [0,«>)  for  each  N,  where  eQ  doesn't  depend 
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£ 

on  N,  and  Y  (0)  converges  in  distribution  to  U(0)  as  e  ■+  0,  we  have 

YEC)  •+  U(0  weakly,  the  unique  solution  to  the  martingale  problem  for  operator 

A  with  Initial  condition  U(0) . 


The  main  burden  of  proof  is  in  finding  the  functions  {fG,N( -) },  and  the 
method  for  this  will  be  developed  in  Theorem  3.  In  our  case,  the  {uE,N(*)}  can 
be  shown  to  be  tight  via  the  method  of  [7],  Theorem  2,  which  also  makes  use  of 
the  functions  {fE,N(*)}. 

_  ^  ce,N  ,  *e,N,e,N,  .  1  e ,N , ,c ,N .  ,  .  ,e ,N ,  , , 

Define  the  operator  A  by  A  f  (en)  =  —  E  [f  (en+e)-f  (en) ] . 

£  n 

A  '  has  the  character  of  an  infinitesimal  operator.  Considering  the  f  as 

£  N 

"test  functions",  the  aim  of  the  theorem  is  to  find  functions  f  '  which  are 

close  to  the  test  functions  and  such  that  the  action  of  A~'N  on  fE,N  is  close  to 
N  3 

the  action  of  (A  +  — )  on  the  test  function  f,  for  each  N. 


0 

III.  Tightness  of  {U^,  small  e,  large  n} 

Assumptions .  Henceforth  we  stick  to  problem  (1.9)  or  (1.10).  Since  we 

treat  the  case  where  the  {£E}  can  depend  on  the  {xE},  some  information  on  the 

n  r  n 

nature  of  the  dependence  must  be  provided.  If  {£E}  were  a  sequence  of  indepen¬ 
dent  random  variables,  then  the  assumptions  and  development  would  be  much  simpler. 
But  it  is  worthwhile  to  do  the  general  case,  since  various  forms  of  it  occur 
frequently.  We  assume  that  there  is  an  auxiliary  process  {£E}  such  that 

U  , ,X  }  is  a  Markov  process  with  a  homogeneous  transition  function,  and  that 
n-  x  n  - * -  - - 

E  *  C  E  £ 

t  is  a  function  of  £  ,  X  .  If  X  were  held  fixed  at  a  value  x  for  all  time, 
n  n  n  n 

then  we  have  a  Markov  process  which  we  denote  by  (£E(x)}  and  whose  transition 

n 

£  £  . 

probabilities  are  defined  by  the  marqinals  of  that  of  {£  . ,X  );  i.e.,  qiven  by 

~ — -  n-i  n 

PtC  (x)Gc||  .  ( x)  ,  x }  =  P{£EEc|£G  ,  =  I  ,  (x),XE  =x} .  Now  a  function  £  (x) 

n  n-l  n  ’  n-1  n-l  n-l  n 


m 


M 


can  be  defined,  where  £  (x)  is  obtained  from  (£  (x),x)  in  the  same  way  that  £ 

n  n  n 

~  0  £ 

was  obtained  from  the  pair  (£^,Xn)  .  Define  the  "partial"  i-step  transition 
function  P{*,  •,•[*}  for  (£n(x) }  by 

(3.1)  P(£.(x) ,4,b|x)  =  P{£.,.(x)E  B|£  (x) ,x}. 

3  J 

The  conditions  introduced  below  will  be  on  the  rate  of  convergence  of  certain  l- 
step  transition  functions  to  their  limits  as  l  ®,  and  on  the  smoothness  of  the 
transition  functions.  Some  smoothness  is  required  in  order  to  "average  out"  the 
discontinuity  in  the  sign  function.  The  symbol  K(x)  will  be  used  to  denote  the  set 
t-k(x)  ,“>)  ,  and  M  denotes  a  constant  whose  value  may  change  from  usage  to  usage. 

We  use  the  following  conditions,  some  in  Theorem  2  and  some  in  Theorem  3. 

(Al)  For  each  x,  there  is  a  unique  measure  P(-|x)  such  that 

oo 

l  | P (£ , £,K(x) | x)  -  P (K(x) | x) |  <  M 

uniformly  in  £  and  in  x  6  [x„,x  ].  P ( * | x)  would  normally  be  the  marginal 

~  U 

of  the  stationary  measure  of 

(A2)  For  each  x,  P( • | x)  has  a  density  p( • | x)  which  is  symmetric  about  £  =  0. 

Define  g(x)  =  /sign [k(x) +£]p(£ | x) d£ .  Let  g ( 9 )  =  0.  g(*)  is  differentiable 

at  x  =  9  with  g^(9)  >  0  and  there  is  a  non-negative,  non-decreasing  func¬ 
tion  q(-)  such  that  g(x)  >_  q(x)  for  x  9  and  g(x)  £  -q(x)  for  x  £  0  and 
q(x)  ^  0  £f  x  i*  9 .  k(-)  is  continuously  differentiable. 

(A3)  P(  ly,®)  |x)  and  P(£,£,  ty ,°°)  |x)  are  continuously  differentiable  in  x,y,  the 


continuity  being  uniform  in  l  >  0,  £  and  in  x,y  in  bounded  sets.  Also  (the 
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| 

i 


r 

I 

i 

i 


; 

f 


p 


subscripts  x,  y  denote  the  derivatives) 

oo 

l  |p  (£,*., K(x) |x)  -  P  (K(x)|x)|  <H, 
t-1 

A 

uniformly  in  £  and  x  £  ^xj,'xu^* 

a  j  A 

(A4)  P(Cn (x)  e  C| 5^^ (x) ,x)  is  continuous  in  x  in  some  neighborhood  of  9 , 

uniformly  in  C  and  £  , (x) . 

- -  -  n-i 

(A5)  P(£,l, t-b,b] |  x)  +  0  as  b  +  0  uniformly  in  £  and  in  x  in  some  neighborhood 

of  0 . 


Let  P^  denote  the  regular  conditional  distribution  of  {£n+j(0),  j>0}  con¬ 
ditioned  on  £  (0)  =  £e  ,  the  actual  sample  value  at  time  n-1.  Let  E0  denote 

n-1  n-1  n 

the  corresponding  regular  conditional  expectation.  We  need  convergence  of  the 
conditional  expectations  of  certain  functions.  In  particular, 


<A6)  There  are  functions  q(m)  which  we  write  as  (which  define  the  expectation 
0 

operator  E  ) 


q(m)  =  Ensign  £Q(0)  sign  £m<0)  =  E0  sign  £^(0)  sign  £jj+m(0) 

such  that 

0 

E  sign  £.  ,  (9)  sign  £.,  .  (0)  ■+  q(m)  as  i  00 
n  l+n  l+n+m  — 

for  all  £E  ,  and  n  >  0.  Also  ,  |q(m) |  <  «  and 
-  n-1  -  -  m=l 

oo  00 

J  £  |E9  sign  £,(0)  sign  £_  ( 9)  -  E  sign  £.(9)  sign  £  ( 0 )  |  <_  M 

j-n  ft-j+1  n  3  *  3  i  ~ 

for  all  £E  , , and  n  >  0 . 


% 


Define 
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eg  (x)  =  [a  (x) l{k(x)+5>0}  -  b  (x)  l{k(x)+S<0})  p(?|x)dC. 

Note  that  g£(-)  has  the  properties  ascribed  to  g ( - )  in  (A2) . 

In  Theorems  2  and  3,  all  0 ( - )  and  o(*)  are  uniform  (in  the  O(-),  o(») 
property)  in  all  variables  other  than  their  argument. 


Theorem  2.  Assume  (Al)  -  (A3)  .  There  is  a  sequence  of  integers  N  ->  °°  as  e  •+  0 


and  an  e^  >  0  such  that  the  double  sequence  (U^,  n>N^,  e£eq}  is  tight. 


Proof .  Part  1.  The  method  is  based  on  combination  of  a  Liapunov  function  and 

2 

an  averaging  technique.  Define  V(x)  =  (x-9)  .  Note  that 


g£(x)  =  -ae(x)P(K(x)  |  x)  +  b£(x)  (l-P(K(x)  |  x) ) 

and 

(3.2)  Ee  V (XG  . )  -  V(XG)  =  V  (XG)EG(XG^.-XG)  +  0(e2), 

n  n+1  n  x  n  n  n+l  n 

(3.3)  EG(XG  -XG)  =  -a  (XG)P(CG  .,1,K(XG) |XG)  +b  (XG)(1-P(|G  ,1,K(XG) |XG)) . 

n  n+l  n  e  n  n-1  n  n  e  n  n-1  n  n 

For  small  E, 

EG(xG-e)[-ac(XG)l{k(x^)+^°}  +  b(XG)l{k(XE)+5E<0}] 

n  n  e  n  n  n —  e  n  n  n 

<  eEE(xG-6)  t-i{k(xG)+5E>0}  +  l(k(XE)+Ce<0}] . 

—  n  n  n  n—  n  n 


Thus 


15. 


EnV(Xn+l)-V(Xn)  ±  EVv()0  11  “  2P(^  ,  ,1.K(X*)  |x*)  ]  +0(e2). 

n  n+i  n  —  x  n  n-1  n  n 

Now,  we  introduce  a  perturbation  vE(n)  to  the  Liapunov  function.  It  is 
used  as  a  technical  device  to  allow  us  to  effectively  replace  the  P  which  appear 
in  (3.3)  by  the  "stationary"  probabilities  introduced  in  (Al) .  Define  vE(n)  by 

00 

V^(n)  =  -2e  l  V  (XE)  [P(CE  , j-n+l,K(XE)  |xE)  -  P(K(XE)|xE)J  =  SE. 
j_n  x  n  n-i  n  n  n  n 

V^(n)  is  well  defined  and  0(e)  by  (Al) .  VE(n)  is  introduced  in  order  to  average 
out  the  noise  Cn_2  in  (3.2).  Using  the  definition  of  g(x),  we  have 


<3‘4)  En+ivi  (n+1)-vi  (n)  =  -V  (XE) [ei(XE)  +  (1-2P(CE  _  ,1,K(XE)  |XE)))  +  T 
**▼1  x  x  x  n  n  n-J.  n  n 


where 


-T 


2cj.LiE"(vx<x"*ilIP<^’J‘n'K<x"*i1|x«i’  ■ 


V  (XE)[P(CE  ,j-n+l,K(XE) |xE)  -  P(K(XE) |xE)]}. 
x  n  n-i  n  n  n  n 


We  show  that  |te|  ■  0(e2)  . 

First,  we  simplify  TE.  Note  that  replacing  XE  in  V  (XE  ,)  by  XG 

n+1  x  n+1  n 

2 

only  alters  the  sum  by  0(e  ) .  By  making  this  replacement,  writing  2V  (XE)  = 

x  n 


6. 


0(e)  and  using  the  Markov  property  (3.5)*,  we  have  the  equivalent  form  (3.6): 


(3.5)  P(Cn_1.  j-n+1,  K(Xn)|x^)  -  EnP(Cn,  j-n,  K(Xn) |Xn> 


(3.6)  -T£»0(e)  l  E^(IP(^,  j-n,  KtX^Jlx^,)  -  P(K(xl,)|xL,)) 

j=n+l 


-  (P (C,  j-n,  K(X  )|x  )  -  P (K (X  )  fx  ))). 
n  n  n  n  n 


By  (A3)  and  the  law  of  the  mean,  (3.6)  equals  (writing  6xe  =  XE  , -XE) 

n  n+l  n 


i. 

0(e)Ee(xe  ,-xe)  l  I  ds(P  (CE,j-n,K(x)  f x)  -P  (K(x)|x)] 
n  n+l  n  J  x  n  x 


where  x  *  XE  +  s6xe  is  used  in  evaluating  the  argument  of  the  integral .  But  the 
n  n 

last  expression  is  0(e2)  by  (A3)  and  the  fact  that  |xE+1-xE|  =  O(e) • 


Part  2.  Define  VE(n)  =  V(XE)  +  VE(n)  and  note  that  VE(n)  *  0(e).  Sum- 
-  n  i  i 


marizing  the  calculations  in  Part  1  yields 


(3.7)  EEVE(n+l)-Ve(n)  <  -  ev  (XE)£  (XE)  +  0(e2), 
n  —  x  n  e  n 


By  (A2)  and  the  fact  that  XE  E  tx£'xu^'  there  is  a  y  >  0  such  that 

(3.8)  EEVE(n+l)-VE(n)  <  -cy  v(XE)  +  0(e2) . 
n  —  n 


♦Recall  that  (£.(XE),  j>n,  1  ,(XE)=£E  . }  is  a  Markov  process  with  initial  condi- 

j  n  n-i  n  n—i 

tion  CE_r 
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Hie  V(XE)  on  the  right  side  of  (3.8)  can  be  replaced  by  VE(n)  without  violating 
n 

£  £  2 

the  inequality.  Consequently,  EV  (n)  <_  (l-ey)EV  (n-1)  +  0{e  ),  which  implies 

that  EVe (n)  »  0(e)  for  large  n?  i.e.,  that  there  is  a  N£  <  e  such  that  VE(n)  = 

0(e)  for  n  >  N  .  Since  VE(n)  =  0(e),  EV(XE)  =  0(e)  for  n  >  N  also,  and  the 
—  e  i  n  —  e 

theorem  is  proved.  Q.E.D. 

Note  on  the  untruncated  {XE}  (x.  =  -®,  x  =  <*>)  .  If  we  used  (1.2)  rather 
-  n  t  u 

than  (1.9),  then  (3.7)  would  still  hold,  and  we 

would  still  have  vE(n)  =  0(e) .  But,  since  (3.8)  would  not  hold,  we  have  not 

resolved  the  problem  of  showing  that  (3.7)  implies  the  existence  of  {Ne)  such 

that  (uE,  n>N  ,  small  e}  is  tight, 
n  —  e 


IV.  The  Limit  Theorem 
Define 

00 

o2  -  1  +  2  £  E9  sign  £o(0)  sign  £  (0) . 
j=l  3 

theorem  3.  Let  {n£}  be  any  sequence  such  that  n£  >_  N£ .  Under  (Al)  -  (A6) , 

{ue(*)»  e  small,  UE(0)  *  UE  }  converges  weakly  to  the  diffusion  U(*)  defined  by 

e 

(4.1)  dU  *  -gx(9)Udt  +  odB. 

Let  n£-Ne  ■  q£.  If^  eq^  •  as  e  -*■  0,  then  the  weak  limit  is  the  stationary 


solution  of  (4.1). 
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Proof.  Part  1.  The  main  work  of  the  proof  is  in  constructing  the  fE  and  veri¬ 
fying  (2.3)  -  (2.5),  where  the  operator  A  is  the  differential  generator  of  (4.1). 

C  C  f! 

Itien  if  {u  (•),  e  small}  is  tight  (where  U  (0)  =  U  ),  the  first  conclusion  will 

n 

e 

follow.  The  tightness  proof  will  follow  readily  from  Theorem  2  of  [7]  or  (11).  The 

assertion  concerning  stationarity  is  not  hard  to  get,  but  the  proof  is  omitted 

owing  to  lack  of  space.  Since  (UE  }  is  tight,  any  subsequence  contains  a  further 

e 

subsequence  which  converges  weakly  (i.e.,  in  distribution).  All  limits  will  be 

of  the  form  (4.1)  and  can  differ  only  in  the  initial  condition  U(0) .  Also,  any 

tight  sequence  (uE(0)}  contains  a  subsequence  which  converges  weakly  to  some 

random  variable.  Thus,  in  view  of  Theorem  2,  we  can,  without  loss  of  generality, 

£ 

assume  that  (U^  }  converges  weakly  to  some  random  variable  U(0) .  Noting  that 
e 

|uE,N|  <_  N,  we  have  XE,N  E  [x  +e,x  -e),  and  6  €  [x  +e,x  -e]  for  small  e. 
n  n  x,  u  x»  u 

Suppose  that  e  is  small  enough  for  the  last  sentence  to  hold.  Then  a  (X  '  )  = 

E  n 

b£(XE,N)  =  e.  So  the  {uE,N}  for  the  truncated  problem  (1.9)  -  (1.10)  are  actually 
given  by  (2.2)  for  small  e,  and  we  use  (2.2)  henceforth . 

^2  3  ^ 

Part  2.  Fix  f (*,♦)£  %  '  which  is  dense  in  as  required  by 

Theorem  1.  We  drop  the  superscripts  N  for  the  sake  of  notational  simplicity,  but 

we  are  actually  working  with  (UE ,N,XE ,N,UE ,N( • ) ,XE ,N( • ) ,ee,N)  and  not  with 

-  n  n  n  - 

(U_,...)  henceforth.  We  have 
n  - 


(4.2)  EEf (UE  . ,n£+E)-f (UE,ne)  =  Ef  (UE,n£)  +  o(e)  +  f  (UE,n£)EE (UE  -UE) 
n  n  +J.  n  t  n  u  n  n  n+1  n 


+  ^  f  (Ue,nE)b*(Ue), 
2  uu  n  N  n 


where 


Wrun>  -  *  v<0  “ 


2P^n  1  #l»K(0+v/e  UE)  |e+/e  UE)  ] . 
n-i  n  n 


Actually,  since  UE'N(0)  =  Un  is  the  initial  condition  for  UE,N(*),  all  indices  n 

E 

should  be  n  +n  and  the  time  argument  n e  should  be  nE-n  £ .  But  we  "shift"  to  the 
origin  for  notational  simplicity. 


19. 


The  third  term  of  (4.2)  must  be  "averaged  out",  and  this  requirement  determines 

the  form  of  the  fe.  We  will  look  for  f£(*)  of  the  form  fe(ne)  =  f(u£,ne)  +  f£(ne) 

-  -  n  u 

+  fE(ne).  For  our  choices,  the  fP  will  all  be  0(f)  and  (2.3)  -  (2.5)  will  hold. 

Define 

00 

f£(ne)  =  2bN(u£)fu(UE,ne)  l  [P(€*_lf  j-n+l,K(X£)  |x£)  -  P(K(X£) |x£)  ] . 

‘  n  d  n  j=n  5 

By  (Al) ,  fE(ne)  is  well  defined.  We  have 


(4.3)  E£  f£(ne+e)-fE(ne)  =  o(e)  -  third  term  of  (4.2) 
n+l  o  u 


+  /e  b  (Ue) f  (U  ,ne)(l  -  2P(K(X  ) |x  )) 
N  n  u  n  n  n 


j=n+l 


w 

+  2/1  l  Eeb„(UE)f  (U£,ne) (P(CE  , , j-n+l,K(XE) |xE)-P(K(XE) |xE) ) . 
,  . .  n  N  n  u  n  n-l  n  n  n  n 

j=n+l 


Let  T£,  T£  denote  the  last  two  terms.  Noting  that  2P(K(x) |x)-l  =  g(x) ,  we  can 
write  the  third  term  on  the  right  of  (4.3)  as 


(4.4)  -  /£*bN(U£)fu(U£,n€)5<X£)  =  -  ebN(U£)fu(UE,ne)ix(6)u£  +  o(e)  . 

Next,  we  simplify  the  T£,  Write  fu*Un+l’nG^  =  bN*Un*  fu^Un'n£*  + 

(b  (Ue)f  (UC,ne))  (UE  ,-U£)  +  0(e),  and  split  tP  into  the  three  corresponding 
N  n  u  n  u  n+l  n 

sums:  Te  ■  t£x  +  T£2  +  TE3.  Now  T£3  =  0(e3/2)  =  o(e).  Noting  (3.5),  combine 

+  T£  and  use  a  differentiability  argument  such  as  used  below  (3.6),  together 
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with  the  fact  that  |xE+^-xE|  £  E>  to  get  that  TE^  +  TE  =  o(e)  .  The  above  sim¬ 
plifications  of  (4.3)  yield 


(4.5)  EE  fE(ne+e)-fE(n  )  =  -eb„(UE)f  (UE,ne)g  (0)UE  +  o(e)  -  third  term  of  (4.2) 
n+i  u  u  Nnun  xn 


2/%'  ’uE>«r<'  (P(«‘.j-n.K(x'+1,|x=il, 

j=n+l 


By  another  differentiability  argument  it  can  be  shown  that  if  the  XE+^  in  the  P 
terms  of  (4.5)  are  replaced  by  8,  then  the  sum  changes  only  by  o(e).  Make  this 
replacement  and  note  that  P(K(0)|0)  =  1/2.  The  revised  sum  in  (4.5)  is 


(4.6)  e(bN(UE)fu(UE,nc))u  £  2EE  sign[k(XE)+EE][P(EE,  j-n,K(0)  |0)  -  +  o(e)  . 

s  u  j=n+l  1  ‘  ' 

e 

Next,  it  is  not  hard  to  see  that  by  dropping  the  k(X  )  term  in  (4.6) ,  the 

n 

sum  changes  only  by  o(e) .  To  see  this,  use  (Al)  and  the  fact  that  |uE|  £  N+l 

and  also  (implied  by  (A5) )  EE  |  sign (kfS+v^eU^)  +£E ]  -  sign(£E)  |  +  0  as  e  +  0, 

uniformly  in  E  .  and  in  X  E  tx„,x  ].  Let  us  drop  this  k(X  )  term.  In  addition, 
n-i  n  *.  u  - * -  n  - 

(A4)  and  (Al)  imply  that  if  the  Ee  in  (4.6)  were  replaced  by  E®  (and  £E,  £E  by 

n  n  n  n 

E  (0),  E  (0))»  then  the  sum  would  change  by  o(e)  only.  This  last  assertion 
n  n 

£ 

follows  from  the  observation  that,  after  the  ktX^)  is  dropped,  the  jth  summand  is 

c  c  a 

actually  EK(sign  E  )E  sign  EjO)-  Then  note  that 
n  n  n+1  3 


E®  sign  V®)  j  J+iEn+1  sign  V®*  "  En  sign  5n  J^n+l  sign  5j(9> 

5  E®F(£  (0))  -  EEF(e3 
n  n  n  n 


where  by  (Al) ,  F(*)  is  uniformly  bounded  in  n,  e.  Now  use  (A4) .  Then  we  can 

write 


(4.7)  (4.6)  =  o(e.;  +  £<b  (U  )f  (U  ,ne))  £  E  sign  £  (0)  sign  £.(0). 

Nnun  u.  .,n  n  j 

3=n+l  J 


Part  3.  We  now  "average  out"  the  sum  in  (4.7).  Define 


00  00 

f^(ne)  =  e%<U^)fu(U^,ne))u  l  l  [E®  sign  £  (0)  sign  £^(0) 

n  j=n  £=j+l  3 


E  sign  £  .  (0)  sign  £.(8)1 
j  * 


The  sum  is  well  defined  by  (A6) .  By  an  argument  similar  to  that  used  in  Part  2, 
we  can  show  that 


EEfE(ne+e)-fE(ne)  =  -  (4.7)  +  o(e) 
n  i  x 


+  e(b  (U  )f  (u  ,ne))  7  E  sign  £  (0)  sign  £.  (0) 

Nnun  u  .  ,  u  3 

3=1 


Finally,  summing  up  the  calculations  and  cancelling  terms  whenever  possible  yields 


|fp(ne)|  +  | fE (ne) j  =o(/e). 


EGfe(ne+e)-fe(nc)  =  o(e)  +  ef.(UErne)  -  eb  (UE) f  (UE,nE)g  (0)UE 
n  tn  Nnun  x  n 


+  2fuu{Un'ne,b^(Un)  +  E  V1#  VUn'nE)  >  u  ^  ^  sign  V6)  sign  5i(9) 

j=l 


( 
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Thus  (2.3)  -  (2.5)  hold  for  an  operator  AN  which  equals  the  operator  A  of  (4.1) 

in  each  S„. 

N 

£  N 

Tightness  of  {U  '  (•),  small  e}.  The  tightness  proof  of  Theorem  2  of  [7]  or  [11] 

£ 

requires  the  construction  of  f  (ne)  similar  to  those  used  here  -  for  each  N.  But 
since  our  f?(ne)  are  O(i^e),  the  conditions  of  the  cited  theorem  hold  and  we  get 
the  tightness  immediately  in  our  case.  Q.E.D. 

£  N 

Remark.  In  more  general  cases  the  f  '  are  chosen  in  a  similar  way: 

e  N 

f  '  =  f  +  small  perturbation.  First  we  obtain  an  expansion  (up  to  o(e))  of 

£  N  £  N  £  #N 

E  '  f(Y  '  ,ne+e)  -f(Y  ,  en)  .  Then  check  which  terms  in  the  expansion  need  to  be 

n  n+1  n 

averaged  out  -  or  replaced  by  an  "average  value".  These  will  be  the  terms  which 

£  N  £  N 

do  not  depend  solely  on  Y  '  ,  en.  Then  the  sum  f„'  is  introduced  (centered  about 
- -  n  o 

a  "mean  value"  -  which  is  the  averaged  replacement  for  the  undesirable  term) . 

.  N 

Continue  as  in  the  proof,  building  up  the  operator  A  step  by  step. 


J 
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